SeCo: Exploring Sequence Supervision for Unsupervised Representation Learning

A steady momentum of innovations and breakthroughs has convincingly pushed the limits unsupervised image representation learning. Compared to static 2D images, video one more dimension (time). The inherent supervision existing in such sequential structure offers a fertile ground for building learning models. In this paper, we compose trilogy exploring basic generic sequence from spatial, spatiotemporal perspectives. We materialize supervisory signals through determining whether pair samples is frame or video, triplet correct temporal order. uniquely regard as foundation contrastive derive particular form named Sequence Contrastive Learning (SeCo). SeCo shows superior results under linear protocol on action recognition (Kinetics), untrimmed activity (ActivityNet) object tracking (OTB-100). More remarkably, demonstrates considerable improvements over recent pre-training techniques, leads accuracy by 2.96% 6.47% against fully-supervised ImageNet task UCF101 HMDB51, respectively. Source code available at https://github.com/YihengZhang-CV/SeCo-Sequence-Contrastive-Learning.